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ABSTRACT 

Guidelines are provided for higher education 
institutions that are developing or evaluating academic review 
activities for their graduate programs. The guide presents the 
critical components of a graduate program review, focusing on the 
philosophy, principles, practices, and policies entailed, and 
discusses the advantages and disadvantages of different review 
methods. The five sections of the guide cover the following areas: 
(1) the purpose of program review; (2) a rationale for program 
* review; (3) a definition of program review; (4) issues which should 
be addressed before beginning the program review, such as locus of 
control, master's versus doctoral programs, relationship of academic 
and professional programs, coordination with accreditation reviews, 
and multidisciplinary and interdisciplinary programs; and (5) the 
structural components of a program review, such as administrative 
support, departmental self-study, use of questionnaires, external 
reviewers, and student participation. Three resources for conducting 
program reviews are described. (LPT) 
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oreword 



Participating in the design and conduct of graduate program reviews is a major 
responsibility of graduate deans. While specific processes used in these reviews may 
differ considerably among institutions, certain basic ideas tend to characterize the 
general approach: it is collegial in the broadest sense of the term and is based on the 
concept of peer review; it is scholarly in that it seeks to define questions whose 
answers will increase understanding of the program; it is comprehensive in that it 
views the program under review as being connected both to other programs wichin 
the university and to the intellectual issues of the discipline at large; and finally, it is 
dynamic in that it results in actions that will improve graduate education. 

This booklet contains a wealth of information and commentary on graduate program 
review, ranging from philosophy and principles to policies and practice. It is 
intended to serve as a guide to institutions that are developing or evaluating their 
program review activities, and to be a useful reference to anyone involved in the 
review of graduate programs. 

Jules B. LaPidus 
President 

Council of Graduate Schools 
Summer 1990 
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ntroduction 



The review of graduate degree programs in the United States and Canada takes place 
in a number of ways and can result in a variety of outcomes. Reviews are conducted 
by individual colleges and universities on their own initiative, by or at the request of 
system offices at the state/provincial levels, or because they are, in effect, mandated 
by state or provincial regulation. They can be simple or complex, inexpensive or 
costly, brief and pointed or endlessly protracted; they can be an integral part of an 
institution's planning and budgeting process, or a token gesture which has no effect 
on future plans. They may be viewed with eagerness as a way to show off stellar 
programs or focus attention on long-neglected needs, or they may be viewed with 
antagonism as useless busy work or a threat to a department's or program's very 
existence. 

Program review is seen differently by many people not because it is unnecessary or 
unimportant, but because it is difficult to do constructively and well. Most faculty 
and administrators agree on the need to evaluate graduate programs periodically in 
some way; if they object, it is usually to the process. If it is to be done at all, program 
review should be done carefully. Otherwise, more harm can come than good: 
institutions and agencies have the impression that there is true accountability where 
there is none; faculty members become disenchanted with the process and withdraw 
their time and support; departmental self-studies turn into "shelf-studies"; programs 
which genuinely need recognition or improvement languish unnoticed; and precious 
institutional and public resources continue to be invested poorly. 

Conversely, when program review is done well, everyone benefits. Faculty and 
administrators gain an understanding of the strengths and weaknesses of their 
graduate programs; then 1 is clear agreement on goals for the future; budget decisions 
are based on academic priorities; institutions are accountable to their students, the 
taxpayers, and other supporters for the quality of their product; and finally, but most 
important, programs improve. Program review moves institutions toward excellence 
in a collegia! way — which is why scholars gather together to form colleges and 
universities in the first place. 

The purpose of this document is not to define a single, correct way to conduct the 
review of graduate programs; that must and should vary by institution, state or 
province. Rather, the intent is to describe the principles of good practice which 
should be present in all program reviews and to discuss the advantages and 
disadvantages of different review methods. If the principles are followed, the process 
will be a successful one, regardless of the method chosen. 
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efinitions 



Because program review at Hie graduate level takes so many different forms, it is 
impossible to describe it in terms that encompass every circumstance and institu- 
tional setting. Therefore, for the purposes of this document, the following terms will 
be used: 

Institution: any organization which engages in program review — most often a college 
or university but also a coordinating agency at the state or provincial level. 

Graduate program: any degree program at the post-baccalaureate level, including 
professional programs. 

Department: the academic and administrative unit in which the graduate program is 
housed and which has ultimate academic responsibility for it. This includes depart- 
ments that are involved with one or more graduate programs, and departments that 
cooperate with interdisciplinary and joint programs. 

Department Chair: the administrative head of the program or department. 

College or Faculty Dean: the dean who has budgetary and personnel responsibility for 
the program to be reviewed, e.g., the dean of arts and sciences, the dean of applied 
science, or the dean of a professional school. 

Graduate Dean: the dean or director of the graduate school or the administrator who 
has primary institutional responsibility for graduate education. 

Academic Vice President: the institution's chief academic officer. In the case of many 
Canadian institutions, this may also be the president/vice-chancellor of the univer- 
sity. 
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urpose of Program Review 



There are many reasons why institutions conduct reviews or participate in provincial 
or itate-wide evaluations of their graduate programs. The primary purpose of all 
program review is the improvement of graduate programs, as measured by the 
quality of the faculty, the students, library and other educational resources, the 
curriculum, available facilities, and the academic reputation of the program among 
its peers. Institutions of higher education, like individuals, require regular scrutiny 
and self-examination to improve, and the systematic review of academic programs is 
an integral part of this process of improvement. In the face of the many external 
pressures on institutions to review programs— from government, system offices, 
public interest groups, and accrediting societies— and the many internal pressures in 
the form of budget adjustments, space needs, and organizational restructuring, it is 
imperative that rhis primary purpose be kept in mind. 

In addition to the improvement of graduate programs, program review, whether at 
the state, provincial, or institutional level, has several associated objectives or goals. 
For the individual university, program review heips in long-range planning and in 
setting both institutional and departmental priorities. It gives administrators and 
academic leaders critical information about the size and stability of a program, its 
future faculty resources and student market, its equipment and space needs, its 
strengths and weaknesses, and its contribution to the mission of the institution. It 
helps set goals and directions for the future, and ensures that overall academic plans 
and budget decisions are based on real information and agreed-upon priorities, not 
vague impressions or theoretical schemes. 

Program review also provides a mechanism for change. Graduate programs, like all 
social structures, evolve slowly; intellectual differences, bureaucracy, time pressures, 
vested interests, concern for survival, and simple inertia all make change difficult. By- 
creating a structured, scheduled opportunity for a program to be examined, program 
review provides a strategy for improvement that is well-reasoned, far-seeing, and as 
apolitical as possible. Changes in graduate programs which are made in the heat of 
the moment or in response to a particular action (e.g., annual budget decisions, 
turnover in adminis trators, individual faculty promotions, student admissions 
decisions, or new course approvals) seldom contain the kind of solid information, 
broad collegia! involvement, and careful thought which a program review promotes, 
and which is necessary for lasting program improvement. 

From an external point of view, program review has two very important purposes. 
First, it provides a mechanism whereby universities are accountable to society for 
their activities and for the quality of their programs. State and provincial govern- 
ments, funding agencies, private donors, taxpayers, and tuition-paying students can 
O ■ ■ 



be reassured through the program review process that the institutions which receive 
their support have graduate programs of high quality which are regularly reviewed 
and revised, and which are responsive to the needs of the society and consistent with 
the aims and objectives of the universities involved. 

Second, program review assists the universities in their efforts to garner financial, 
philosophical, and political support from state or provincial legislatures, state and 
federal funding and regulatory agencies, coordinating boards, and other constituen- 
cies. The information gathered in the review process, and the assessment of program 
strengths and needs, provide strong and compelling evidence of the quality of 
graduate programs, the areas of greatest need, and the foundation on which future 
improvements should be built. This information can and should support external 
decisions about resource allocation, enrollments, special initiatives, research grants, 
and even private gifts. The stronger and more careful the program review process, 
the more persuasive the results. 
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hy Have Program Review? 



Graduate education is replete with evaluations. Faculty are evaluated for salary 
increases, for promotion and tenure and, in many institutions, for membership in the 
graduate faculty; students are evaluated for admissions, performance on compre- 
hensive examinations, and degree completion; courses are evaluated as they are 
added to the curriculum; and facilities and financial resources are scrutinized 
annually in the budgeting process. Program review, however, provides the only 
comprehensive evaluation of an entire academic program, integrating all of the 
elements which contribute to its success. 

While it is true that the reviews conducted by professional licensing or accrediting 
associations are also comprehensive in scope, they have special goals which may or 
may not coincide with those of the institution. Accreditation reviews often are 
extremely focused on the existence of certain minimum standards adequate for 
licensure or accreditation. They do not necessarily contain the broad academic 
judgments and recommendations for change in program direction which should 
come out of a program review. 

Graduate programs are dynamic; they change constantly as faculty come and go, the 
student applicant pool increases or declines, degree requirements are eased or 
tightened, and as the academic discipline just naturally evolves. Although degree 
programs are usually reviewed carefully when they are first proposed, once they are 
approved they ma> never be evaluated again. Constant scrutiny is unhealthy for any 
program, but periodic, thorough review will ensure that the program has lived up to 
its original goals and will identify key areas in which it should be strengthened. It 
will also, if necessary, identify programs which should be cut back or terminated. 

In addition to these intrinsic reasons, some states and provinces have initiated 
periodic program review as part of a legislative mandate or provision by a governing 
board or commission. These reviews may be conducted by the institution itself, by an 
external agency funded by the government, or by a special body established by the 
universities. In such cases, the intrinsic justification for program review is supple- 
mented by real external requirements which may influence the frequency and 
structure of the process. 

Academic program review is therefore an elaborate and often costly process, but it is 
a necessary part of ensuring the continuing quality of graduate programs and of 
identifying ways to improve them. There is no adequate substitute. 




hat Is Program Review? 



Program review may take many different shapes and forms, but it always has certain 
key characteristics. 

1. In the United States the process is usually internal, initiated and administered by 
the universities themselves. It may be conducted at the state-wide or system-wide 
level; most frequently, however, state agencies mandate a requirement for program 
review but leave the actual review to the institutions. In Canada, where the provinces 
are constitutionally responsible for education, including post-secondary education, 
t/ere is considerable variation. In one province— Ontario— all graduate programs are 
reviewed regularly in a seven-year cycle by a central organization (the Ontario 
Council on Graduate Studies), which is administered and funded on a cooperative 
basis by the fifteen provincially-supported universities. Similarly, the Conference des 
Recteurs et Principaux des Universites du Quebec (CREPUQ) is responsible for 
reviewing new graduate programs in its jurisdiction. Program review in the other 
provinces tends to occur at the level of the individual institution. 

2. Program review is evaluative, not just descriptive. More than the compilation of 
data on a particular graduate program, it requires academic judgments about the 
quality of the program and adequacy of its resources. It goes beyond an assessment 
of minimum standards to subjective evaluations of quality by peers and recognized 
experts in the field. 

3. Review of graduate programs is forward-looking; it is directed toward improve- 
ment of the program, not simply assessment of its current status. It makes specific 
recommendations for changes which need to be made in the future, as part of 
departmental and institutional long-range plans. 

4. Departments engaged in program review are evaluated using academic criteria, 
not financial or political ones. They are scrutinized on the basis of their academic 
strengths and weaknesses, not their ability to produce funds for the institution or 
generate development for the state or province. Finances and organizational issues 
are certainly relevant in the review, but only as they affect the quality of the academic 
program (e.g., low faculty salaries, lack of laboratory equipment, rapid turnover in 
department chairs). 

5. To the extent possible, program review is an objective process. It asks graduate 
departments to engage in self-studies which assess, as objectively as possible, their 
own programs. It brings in faculty members from other departments and often from 
outside the institution to review the self-studies and to make their own evaluations, 
using independent judgments. It is part of an established, public process in which all 
graduate programs are similarly reviewed, so the tendency to use a review for 
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political purposes— to highlight only the best programs or punish the worst— is 
minimized. Although no such process can be completely objective, a well-constructed 
program review leads to a careful, arm's-length evaluation by at least some 
individuals with no vested interest in the results. 

6. Program review is an independent process, separate from any other review. 
Reviews conducted by regional or professional accrediting associations, licensing 
agencies, or budget committees are separate and distinct, and cannot substitute for 
program reviews. Data collection and parts of the departmental self-study may often 
serve a number of review purposes, and there is much to be saved in time and effort 
by timing a program review to coincide with an accreditation or other external 
review, if possible. However, to be effective, program review must be a unique, 
identifiable process, which stands on its own, draws its own set of conclusions, and 
directs its recommendations to the only individuals who have the power to improve 
graduate programs: the faculty and administrators of the institution. 

7. Most important of all, program review results in action. Growing out of the 
reviewers' comments and recommendations, the institution develops a plan to 
implement the desired changes on a specific, agreed-upon timetable. This plan is 
linked to the institution's budget and planning process, to help ensure that recom- 
mended changes actually get made, that necessary resources are set aside, and that 
the program's goals fit into the institution's overall academic plans. If no action 
results from the review, departments soon lose interest in the process, the quality of 
the product deteriorates rapidly, and large amounts of time and money are wasted. 
In addition, other less objective and collegial ways of making decisions arise, and the 
advantages of systemat* \ io;ram review are lost. 

Successful program review, : Iv n, is a process of evaluation which has all of the above 
characteristics. It provides answers to the following kinds of questions: 

• Is the department advancing the state of the discipline or profession? 

• Is the teaching or training of students useful and effective? 

• Does the program meet the institution's goals? 

• Does it respond to the profession's needs? 

• How is it assessed by experts in. the field? 

Clearly, this list of questions can be supplemented by others, and the emphasis given 
to any particular question depends on the mission of the institution and the 
individual graduate program. But these are the kinds of questions that program 
review is designed to address. 




ssues to be Addressed Before 
Beginning 



There are a number of issues that need to be resolved before the first review begins. 
They have to do with how the reviews will be conducted, their scope and timing, 
who will be involved in the process, and the relationship with other review activities. 
While there is no single right way to conduct a program review, there are advantages 
and disadvantages to each of the approaches described below, depending on the 
institutional setting. 

Graduate versus Total Program Review 

Where graduate and undergraduate degree programs exist in the same department, 
institutions differ significantly in whether they choose to review graduate programs 
alone or as part of an overall review, including undergraduate programs. There are 
clear advantages to each. 

Reviewing graduate programs alone allows for a careful, thorough, in-depth review. 
With the narrower focus of just graduate education, both the department in its 
self-study and the review committee in its report can give more attention to those 
quality indicators which are unique to graduate education: content of graduate 
seminars, quality of the research experience, productivity and professional involve- 
ment of graduate students, thesis load of faculty members, and professional 
contribution of graduates. Reviewers can be chosen who have special expertise and 
experience at the graduate level, and they ra i reasonably be asked to spend more 
time ferreting out answers to those subjective questions about program quality found 
only through personal interviews with faculty and graduate students. 

Since graduate programs tend to be smaller than undergraduate ones, they easily get 
lost in the sheer numbers of enrollments, faculty teaching hours, and space and 
equipment needs, unless they are reviewed separately. And the questions that are 
asked, if focused primarily on undergraduate issues, are often irrelevant to or paint 
an extremely inaccurate picture of the graduate program. By reviewing graduate 
programs separately, institutions are forced to devote time and in-depth attention to 
them, and to design criteria for evaluation which are uniquely suited to the graduate 
level. A separate process ensures that graduate programs will not be viewed simply 
as costly extensions of baccalaureate study. 

On the other hand, there are advantages to reviewing graduate and undergraduate 
programs together. There is enough duplication of questions and data (e.g., faculty 
qualifications, space and facilities, departmental organization, focus of the discipline) 
to justify combining the reviews into one. A single review can produce significant 
savings in time and money, and create immense good will by not subjecting 
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departments to constant examination. A combined process may be more efficient, 
since in most cases the same faculty will prepare the self-study and the same 
reviewers can be asked to examine both the graduate and undergraduate programs. 
Also, the two are clearly interdependent; matters like faculty teaching loads, program 
and departmental budgets, facilities, and the quality of the teaching assistant 
experience cannot be properly assessed without information on both undergraduate 
and graduate programs. 

Whether graduate programs are reviewed separately or in conjunction with under- 
graduate programs, two basic principles apply. First, graduate programs must be 
looked at with a separate set of standards and criteria, and by those who have an 
understanding of the unique demands of graduate education. Undergraduate work 
involves breadth; graduate work, focus. Excellence in one in no way guarantees 
excellence in the other. Second, if undergraduate programs are also present in the 
department, they must be taken into account even in graduate-only reviews. For 
while the ' vo are different, they are seldom, if ever, unrelated. They are part of the 
same organic unit — the academic department — and both draw their strength from 
the same faculty source. 

Locus of Control 

Who conducts the program review depends largely on whether the review is 
intensity controlled, or conducted externally as part of an ongoing cycle of reviews. 
When the process is internal, involving graduate and /or undergraduate work in an 
entire department or school, it is usually conducted by the academic vice president's 
office. A university- or college-wide faculty panel is often appointed to establish 
criteria and procedures for the review and to receive any final recommendations. 
That panel may conduct the review itself or appoint a subcommittee, depending on 
the size and complexity of the institution, the number of reviews scheduled, and the 
expertise needed. Clearly, in this model, panel and subcommittee members need to 
be competent to judge both graduate and undergraduate programs. 

When graduate programs are reviewed internally but separately from undergradu- 
ate programs, such reviews are often initiated and conducted by the graduate dean. 
He or she may use the standing graduate council for this process or, like the academic 
vice president, appoint special review committees. After the process is completed, 
recommendations then come back to the graduate dean and the graduate council, for 
recommendation to the college or faculty dean and the vice president. 

If the review process is undertaken externally, it is usual for the scope to be restricted 
to the graduate level. In this case, the external reviews are based on a common set of 
information provided by each program, and the review is undertaken by a commit- 
tee representing the collectivity of universities. This committee may seek the advice 
of expert consultants if required. The graduate dean acts as the crucial point of 
contact between the external appraisal committee and the university. 

Regardless of who conducts the review process, the following principles apply: 
O 



• Whether the review is of the graduate program only or an entire academic 
department, all reviews should involve the college or faculty dean. 

• The graduate dean should play a major leadership role in all reviews, either 
as initiator or key participant. 

• The essential participants in any graduate program review are: the academic 
vice president, college or faculty dean, graduate dean, department chair, 
departmental faculty, review committee(s), and graduate students in the 
program. 

Without the involvement of all of the above constituents, the process cannot succeed. 

Master's versus Doctoral Programs 

In those institutions with research-oriented master's and doctoral programs in the 
same department (e.g., English, economics, and physics), there is no question that 
both should be reviewed simultaneously. The similarities in curriculum, faculty, 
research facilities, and often, student body, would make separate reviews illogical 
and costly. 

However, practice-oriented master's programs which do not tie directly to doctoral 
work (e.g., the M.B. A., M.S.W., or M.Ed.) deserve careful, separate attention. Whether 
it leads to a doctoral program or not, a master's degree should have its own academic 
integrity; it should be seen neither as simply a step toward the doctoral program nor 
a continuation of undergraduate work. A master's program may have a different 
student clientele and different degree requirements, place graduates in different 
kinds of positions, and even involve different faculty than the doctoral program in 
the same department. The institution should examine the unique characteristics of 
each such master's program, and develop criteria for evaluation that are appropriate 
for that program. 

Relationship of Academic and Professional Programs 

Traditional academic (research) programs and professional (practice) programs often 
exist within the same graduate departments, e.g., business, education, and engineer- 
ing. Professional programs typically have very different educational goals from 
academic programs and are often reviewed independently for accreditation. Should 
institutions attempt to review such programs together? Do professional programs 
which are subject to separate accreditation review need institutional review as well? 

Although they may have separate student bodies, separate degree requirements, and 
sometimes separate faculties, academic and professional programs within the same 
academic unit should be reviewed together. They are, by definition, part of the same 
discipline, or grow out of the same disciplinary tradition; they are housed adminis- 
tratively and budgetanly in the same unit; and they necessarily draw from each 
other's resources to complement their intellectual programs, Like graduate and 
undergraduate programs, they are separate but interdependent, and budget, facili- 
ties, and faculty appointment issues which affect one almost always affect the other. 
© 



It is essential, of course, that they be reviewed using different criteria since their 
missions and standards of excellence are inevitably different. To use the same criteria 
is to do both a disservice. 

Given these differences, it is tempting to substitute professional accreditation reviews 
for internal review of professional programs: the criteria are nationally recognized 
and established by the profession; the reviewers are external to the institution and 
experts in their field, and the process is usually careful and thorough. However, a 
professional program which meets national accreditation standards does not neces- 
sarily meet the institution's criteria for excellence or conform to its mission. A 
program could be superior by national standards but completely inappropriate for 
the local community, or an excessive drain on resources needed for research. 
Conversely, it might not meet minimum accreditation standards on such matters as 
faculty-student ratio but tie closely to the overall mission of the institution and satisfy 
that institution's quality standards In either case, institutions which have profes- 
sional programs need to review them in the same fashion as other graduate programs 
and not delegate that responsibility to an outside agency or organization. 

Coordination with Accreditation Reviews 

While program reviews need to be separate processes from accreditation reviews, it 
is possible and desirable to juxtapose them. The two can seldom be held simulta- 
neously, but much is to be gained by conducting both reviews during the same 
academic year. Many elements of the departmental self-study and of the data 
collected are the same, as are, of course, such items as faculty curriculum vitae; thus, 
significant time, paper, and effort can usually be saved by combining these portions 
of the reviews. Also, in the case of institutionally-based reviews, a campus review 
committee may find the report of an accreditation or other external team extremely 
valuable as it considers its own recommendations. In the case of some state-wide or 
system-wide reviews, the internal review may be required to be conducted before or 
at the same time. 

When both kinds of reviews are conducted simultaneously or at about the same time, 
the question often arises, is it appropriate to use members of a professional 
association's aa, editing team as external reviewers or external members of a review 
committee? In almost all cases, the answer is no, since the two reviews are for 
different purposes and use different criteria for evaluation. In those rare cases where 
it is done, special care should be taken to ensure that these individuals have the same 
qualifications as those who would be appointed as external reviewers anyway, i.e., 
they have experience with and knowledge of graduate programs, and that the two 
processes — program review and accreditation review — are kept separate. 

Scheduling the Reviews 

The scheduling of program reviews is critical; more well-meaning plans for compre- 
hensive program review have foundered on an unworkable timetable than on any 
other obstacle. The frequency of reviews will, of course, vary significantly, depending 
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upon the institution's size and complexity. However, a cycle of review of every 
graduate degree program or department every five to seven years is recommended. 
To do so more often is to create an unmanageable process and risk over-evaluation; 
to do so less often is to lose track of the content and quality of one's graduate 
programs. The intensity of the reviews may vary; e.g., each department might have 
a major review every seven to ten years with an update every three to five. What is 
essential is that the reviews be cyclical and that they be conducted on a known 
timetable which continues indefinitely into the future. They should also be incre- 
mental with each review building on the previous cycle's report. 

A reasonable length of time for the completion of each review is twelve to eighteen 
months. It generally takes at least a year to complete the self-study, committee review, 
and preparation of a final report, with the process sometimes extending into the next 
academic year. There are, of course, no hard and fast rules in this area; each 
institution must establish a calendar which is reasonable and realistic for its 
circumstances. 

Any nuinoer of unexpected obstacles may delay the piiniss: Hie department chair 
changes, institutional data are incorrect, the review committee disagrees on its 
recommendations, or the external reviewers are unavailable. Since the purpose of the 
review is to improve the graduate program, however, and not just to go through the 
process, delays in the schedule may be necessary and perfectly appropriate. A report 
which is submitted on time but which sits on a shelf unused is of little value. 

At the same time, it is important for a program review not to become too protracted; 
when it does, the information will become outdated and the department indifferent 
to the result. A process which cannot produce results in a timely way will lose its 
credibility and therefore its effectiveness. 

The order in which programs are reviewed is also important. One factor to consider 
is the quality of the programs. While there is a natural tendency to want to review all 
of the perceived poor-quality programs first, this temptation should be resisted, lest 
program review be seen as a veiled method for eliminating or punishing programs 
in trouble. Conversely, all the high quality programs should not be reviewed first 
either, to avoid the impression that the process is nothing more than a whitewash or 
an attempt to spotlight excellence. For the process to be taken seriously as a fair and 
equitable procedure, a judicious mix of programs should be selected, based on 
perceived strengths and weaknesses, for each round of reviews. 

Other factors to consider in the choice of programs are: length of time since the last 
review; compelling financial problems or resource needs; major proposals for 
curriculum changes; upcoming accreditation or other external reviews; and the 
desire to have a program reviewed. 

At smaller institutions, it may be possible to review programs by administrative or 
budgetary unit, e.g., all departments in the humanities or in the school of engineer- 
ing. This has the clear advantage of allowing review of similar programs simulta- 
neously and can help relate the recommendations more directly to that unit's budget 
allocation process. However, at many institutions, such a plan would be unwieldy. 
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Multi- and Interdisciplinary Programs 

Graduate programs which draw on multiple disciplines or are truly interdisciplinary 
pose special problems for review. Ideally, all graduate degree programs should be 
reviewed as individual programs, rather thin through departments, since the two are 
not synonymous. However, given the organization of most universities and colleges, 
the faculty who teach in such programs and the students who study in them are 
almost always arranged into academic departments — units which control faculty 
hiring, student admission, budget, course offerings, and all other elements of a 
graduate degree. 

In spite of the administrative convenience of working through existing departments, 
however, multi- and interdisciplinary programs need to be reviewed separately. Data 
and resources should be gathered from all relevant departments, but put together in 
a single, coherent document. Where possible, much can be gained by reviewing, as 
a part of the same cycle, one or more departments affiliated with the interdisciplinary 
program, since they clearly are interdependent. For example, an interdisciplinary 
program in film and literature might well be reviewed at the same time as cinema 
and English, or women's studies along with history or sociology. Similarly, programs 
in related departments which have a strong intellectual affiliation — even if not 
strictly interdisciplinary — can benefit from simultaneous review, e.g., molecular 
biology, genetics, and biochemistry. 




tructural Components of a Program 
Review 



Although the precise nature of a program review will necessarily vary from one 
institution to another, a number of key elements should be present in all reviews. 
They are discussed below in roughly chronological order. 

Development of the Review Plan 

Before any actual reviews are conducted, the institution, system or other body 
conducting the review should develop, through established decision-making pro- 
cesses, a plan for program review. This document should describe the purpose of the 
reviews, the process which will be followed, guidelines for materials to be included, 
a proposed timetable for reviews, what will be done with the results, and possibly the 
list of departments or programs that will be reviewed each year. A clear plan, which 
is developed jointly by all constituencies and is disseminated broadlyin advance of 
the first review, will help ensure a fair process, with as few surprises as possible. 
Regardless of the type of process followed, all reviews should be based upon these 
principles: 

• The reviev should be designed and conducted by the faculty. 

• The graduate dean should play a major leadership role in the review process. 

• The process should be as open as possible, consistent with the requirement 
that the rights of individuals be protected. 

• The administration should be involved actively and in a positive way in each 
review. 

Administrative Support 

Although often overlooked, adequate staffing, computer resources, and general 
administrative support are necessary to the success of any program review. Program 
review is not free; faculty and administrators must devote significant blocks of time, 
secretarial and other support staff must be identified, space and office supplies must 
be provided, computer time and other institutional research services must be 
committed, and funds must be set aside to pay external reviewers, if they are used. 

Staff and general administrative personnel should be at a high enough professional 
level to deal with the complexity and sensitivity of the task: e.g., to contact external 
reviewers, coordinate site visits, serve as a resource person for the department and 
the review committees, initiate and explain requests for institutional research, and 
generally keep the process moving. Staff support should be provided centrally, not in 



each academic department, in order to keep the findings confidential during the 
review and to avoid overloading departmental staff, thereby alienating departmental 
faculty and slowing down the process significantly. While it is appropriate for the 
department to provide its own staff support for the self-study, the larger review 
process should be staffed centrally. 

Accurate institutional data are critical to the success of any review. Data on academic 
programs should be developed and maintained centrally wherever possible, but 
reviewed and evaluated by the faculty of the program based on their knowledge and 
experience. At many institutions, there is a central office that collects and maintaias 
data on academic programs and which can produce the information needed for a 
program review, such as admissions, enrollments, sponsored research activity, and 
number and type of faculty. Where such an office exists, it should be brought into the 
process early to ensure that it can and will produce the desired reports in a timely 
fashion, and that the desired data elements are indeed available. A standard report 
format using a single set of definitions should be developed in advance of the first 
review in order that each individual report can be generated rapidly and that the data 
are consistent across programs. 

Where data are maintained in various offices for different purposes, e.g., admissions, 
faculty records, contracts and grants, information will have to be requested sepa- 
rately — but again, using a standard, agreed-upon format. Where there are no central 
data services available, the university should consider creating a database strictly for 
program review purposes, or at a minimum designing a standard format and set of 
definitions for departments to use in reporting their own data. The best information 
on graduate programs most often comes from a combination of central and 
departmental sources; either source alone runs the risk of drawing a distorted 
picture. 

In addition to personnel time and research support, funds will be needed for a 
number of direct expenses: office supplies, honoraria and travel costs for external 
reviewers, luncheon and other meeting expenses, computer programming, and 
possibly computer time. Few organizations can cover the cost of thorough program 
review without specific budget allocations for this purpose. 

Departmental Self-study 

Once the administrative mechanisms are in place, the first step in a program review 
is almost always a departmental or program self-study. The self-study is prepared by 
the faculty of the department and is both descriptive and evaluative; it provides basic 
information on the nature of the program and gives the faculty's assessment of the 
program's strengths and weaknesses. It should be clearly written and candid, and 
should adhere to a standard format and agreed-upon definitions for at least the data 
portion of the presentation. 

A departmental self-study is the department's opportunity to scrutinize itself, to 
publicize its accomplishments and examine its flaws; it is also a chance — perhaps the 



only one — to explain itself and to demonstrate how it is viewed by its peers. A self- 
study should answer the following five questions: 

• What do you do? 

• Why do you do it? 

• How well do you do it, and who thinks so? 

• What difference does it make whether you do it or not? 

• How well does what you do relate to why you say you do it? 

In order to answer these questions, the information described below should be 
included, in one form or another, in every self-study While this is by no means a 
comprehensive list, it covers the major areas to be addressed. Wherever possible, data 
should be provided for the previous five years. 

Departmental mission and organization: purpose of the department; contribution to 
the institution's mission; departmental policies and organization, 

Program purfxise: intellectual place in the discipline; goals and objectives of the 
graduate program and related research; national and local need for the program. 

Departmental size: graduate faculty; support staff; master's students; doctoral 
students; degrees awarded; instructional and general expense budget; portion of 
budget used for graduate program(s); amount of externally-funded research; size of 
other externally-generated funds (e.g., gifts or contracts). 

Faculty profile: total number (full/part-time, visiting, tenure/ non-tenure track); 
number of graduate faculty; number of new and retiring full-time faculty during the 
past five years; average age; sex; ethnicity; percent tenured. 

faculty research and scholarship activity: description of faculty research, scholarship, 
or creative activity; individual productivity; external grants submitted and funded. 

Faculty contribution to graduate program: faculty/ graduate student ratio; average 
course load; average thesis load per faculty member and distribution across 
department; distribution of grades in graduate courses; teaching evaluations. 

Student profile: admissions criteria; number of applicants and admitted students; 
actual enrollments (pre- and post-candidacy for doctoral programs); citizenship, 
average age; sex; ethnicity; part-time/full-time status. 

Financial support for graduate students: departmental and institutional funds; percent 
of students on financial aid; average level of support; ratio of grant-to-loan funds; 
number of teaching and research assistantships and selection process. 

Facilities: space (classroom, research, office); laboratory and instructional equip- 
ment; library; computer resources. 

Curriculum: degree requirements; program structure; current graduate courses; 
frequency of course offerings. 

Student productivity: number of theses and dissertations produced in last five years; 
sample dissertation and thesis quality; student publications, exhibitions, and profes- 
sional presentations; degree completion rates; average time to degree completion. 
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Programmatic climate: scholarly community; quality of academic advisement; esprit 
de corps; critical mass of faculty and students; activities related to promoting 
diversity among students and faculty. 

Collateral support: interactions with other departments or units that strengthen the 
program. 

Profile of graduates: number of graduates; job placements; continued contributions 
to profession or field. 

Overall assessment of program: strengths; weaknesses; reputation in the field; goals 
for improvement. 

Use of Questionnaires 

Questionnaires completed by students, faculty and program alumni supplement and 
validate the information and opinions presented in the self-study. A questionnaire to 
all or a selected sample of current graduate students in the program can elicit 
important information on their perceptions of the faculty, curriculum, and overall 
value of the experience. Similarly, individual faculty impressions of the program and 
of its areas of strength and weakness can paint a picture which statistics alone cannot 
provide. Finally, a questionnaire to recent alumni of the program may be especially 
valuable; they are the most likely of the three groups to be objective in their 
assessments, since they are removed from the program by a few years and have no 
more vested interest in it. For professional programs especially, alumni and employ- 
ers of graduates are the best way to assess the usefulness of the degree in the 
profession. If questionnaires are not used, external reviewers should be asked to meet 
with the faculty and graduate students (a good idea in any case). 

To be informative, the responses to all questionnaires should be anonymous. In 
addition, they should be designed with care and with enough attention to logistics to 
ensure success. A number of factors should be considered in determining whether to 
include one or more questionnaires in the review process: the time required to 
develop and distribute them and wait for the responses; the likely response rate* the 
additional paperwork burden on the respondents and the department or review 
committee; the unique information to be gained; and the expense. 

If used, questionnaires should not duplicate information available elsewhere (except 
for purposes of validation), but should concentrate on that unique knowledge which 
can be gained only through this method: e.g., student opinions of the quality of the 
examination or thesis experience, faculty assessments of the overall quality of the 
research effort, and alumni opinions on which aspects of the program were the most 
challenging and effective. These opinions will cither validate or raise important 
questions about the statistics reported by the department itself and point toward very 
specific areas for improvement. 

In some cases, departments present the results of questionnaires as part of the 
self-study; however, it is usually pn^ferable to have such surveys conducted at the 
institution-wide level — in order to provide adequate staff and financial support and 
to ensure the confidentiality and objectivity of the process. 
9 



Institutions may wont to design their own questionnaires or, as an alternative, use 
those already prepared for this purpose. Under the sponsorship of the Council of 
Graduate Schools and the Graduate Record Examinations Board, e.g., the Educa- 
tional Testing Service (ETS) has developed the Graduate Program Self- Assessment 
Service (GPSAS) to assist institutions with program review. The Service sells sets of 
standardized questionnaires for current students, current faculty, and alumni which 
can be analyzed by the institution or by ETS. Questionnaires are available for master's 
or doctoral programs, and institutions may add specific questions of their own before 
the questionnaires go out. Such a service cannot substitute for an institution's own 
investigation and analysis, but it can certainly complement them. 

Review Committee 

At least one committee should be established to review the departmental self-study 
and the results of any questionnaires, and to conduct its own investigation and 
interviews/ as needed. There may be one committee to review the program and 
another (standing) committee to coordinate the process and receive the reports of the 
review committee and of the department. Alternatively, one committee may perform 
both functions. 

In either case, the committee reviewing the department should be composed 
primarily of faculty members who are removed enough from it to avoid conflict of 
interest. Expertise in the field is often left to reviewers brought in from outside the 
institution. Students are sometimes appointed to serve on review committees, though 
more often they are members of standing committees that govern the review process 
and receive the review committee's report. 

The review committee typically reads the departmental self-study, and may conduct 
a one- to two-day site visit in the department, interviewing faculty and students, 
touring facilities, examining sample student files and dissertations or theses, meeting 
with the department chair, etc. They may also review any additional materials which 
the process has elicited (e.g., the results of questionnaires) and interview the relevant 
deans. Alternatively, external consultants may conduct the site visit and report their 
findings to the committee, 

When the review is complete, the review committee usually prepares a succinct 
report on its findings, including recommendations for changes or enhancements. 
This report is most often confidential, except to the department appropriate deans, 
and any relevant standing committee. 

Departmental Response 

Soon after the committee report is submitted, the department should be given a copy 
and asked to prepare a written response. This provides an opportunity to correct any 
factual errors and to reply to specific criticisms or conclusions which the review 
committee has made. Except for factual errors, the review committee may choose not 
to change its report, but it is essential to keep the department knowledgeable about 
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the findings and give them an opportunity to comment on the evaluation. If there are 
multiple reports at this stage, the department should have an opportunity to 
comment on each report or each version. 

External Reviewers 

External reviewers are used in a number of different ways. Some review procedures 
include one or more external reviewers on the internal review committee; others 
establish an entirely separate external review team; others appoint individual experts 
to conduct independent evaluations rather than to work as a team; still others consult 
external reviewers only in rare cases where special expertise is needed which is not 
available on campus. Most program reviews, however, involve external reviewers in 
the process in some way. 

The purpose of including external reviewers in the review process is to provide input 
from recognized experts in the discipline, to ensure the objectivity of the process, and 
to determine where the program fits in the discipline regionally, nationally or 
internationally. It is difficult to accomplish these goals using only faculty members 
from the institution involved, even though the process may be a thorough and 
careful one. External reviewers also lend credence to the final report by contributing 
their acknowledged expertise and broader point of view. They are, of course, costly 
to include, and their intellectual orientation and priorities may not necessarily match 
those of the program being reviewed. 

Where external reviewers are appointed as members of the regular review commit- 
tee, they should be incorporated fully into the process. Where they are brought in 
after or in addition to an internal review, they should be asked to focus on specific 
aspects of the program or to answer specific questions, in order to take best 
advantage of their expertise and not to duplicate the entire review process. 

Individuals chosen to be external reviewers can be identified in a number of ways. 
Most commonly, the department or program in question is asked to submit a list of 
proposed reviewers — faculty members from other institutions who are well re- 
spected in the field but without direct connections to the program in question. They 
should be "at arm's length" from the program, i.e., not graduates or prior faculty 
members and not research collaborators, though they may very likely know some of 
the faculty in the program and be familiar with their research or professional 
activities. 

Reviewers may also be selected independently by the college or graduate dean, or 
the administrative officer responsible for the review, by contacting professional 
assentations or academic deans and department chairs at institutions with strong 
reputations in the field. Outside reviewers should be paid an honorarium for their 
time, plus expenses. 

Student Participation 

There is great variation in the degree of student participation in graduate program 
review. In some cases, graduate students may contribute to the self-study, be full 




members of the review committees, and serve on the graduate council or standing 
committee or panel which makes the final recommendations to the deans and the 
academic vice president. In other cases, they may have nothing to do with the 
process, except possibly through student evaluations of graduate courses and 
meetings or interviews conducted by the review team. 

Graduate students should participate in the program review process. They should be 
asked to complete confidential questionnaires where feasible, they should be 
interviewed collectively and individually by the review committee, and they should 
participate in the departmental self-study. At the institution-wide level, they should 
be involved in the entire program review process. Where they are also appointed to 
serve on review teams or standing panels, they should be full voting members with 
access to all relevant documents. 

Final Report and Recommendations 

After the review committee completes its work and the department has had an 
opportunity to comment, a final report with recommendations should be submitted 
to the academic vice president and /or graduate dean, with copies to all relevant 
parties. This report should present the findings of the review committee in light of the 
self-study and the department's responses to subsequent reports, and make specific 
recommendations for change. Even though there may have been multiple documents 
and reports along the way, the program review procpss should culminate in the 
preparation of a single, comprehensive report, which can be used to develop a plan 
of action. 

Confidentiality 

There seems to be general agreement that some level of confidentiality should apply, 
particularly to interim reports that arise during the program review. This is usually 
taken to mean that the reports are for internal use in the university, available only to 
those who are directly involved in the review 

If possible, the final report and recommendations should be treated in the same way, 
but since the consequences of these reports, e.g., personnel or program changes, or 
space reallocations, are obviously public, the reports and recommendations upon 
which they are based may also become a matter of wider interest. In addition, some 
program review processes, particularly those established by agencies or organiza- 
tions external to the university, may mandate certain kinds of distribution for reports. 
In all cases, some balance should be sought between assurances of confidentiality 
necessary to encourage the utmost candor from participants, and assurances of 
openness needed to ensure that those affected by the review will understand the 
basis for actions taken. 

Implementation 

Program review does not end with the submission of a final report; indeed, the most 
important step still lies ahead. Since the purpose of program review is not to produce 



a report but to improve graduate programs, implementing the recommendations in 
a timely way is essential. Turning recommendations into action involves at least the 
following steps: 

a. One or more meetings of the department chair, college or faculty dean, graduate 
di in, and academic vice president to discuss the recommendations; 

>. A memorandum of understanding drawn up by the above about what specific 
act, ns are to be taken by whom, and by what deadlines; 

c. scussions of the recommendations with the program faculty tor understand- 
ing ai implementation; 

d. In ^ration of the decisions reached into the institution's long-range academic 
planning »nd budget processes. 

These stej s can be accomplished only through a concerted effort by all of the relevant 
parties, as well as budget officers, admissions directors, and facilities planners, whose 
cooperation will be essential for success. The proposed timetable for implementation 
of each action should be realistic, but not so far into the future as to be irrelevant. 
Goals for the department to accomplish before the next program review are also in 
order. 

Once the memorandum of understanding is in place, the academic vice president or 
graduate dean should respond to the review committee, explaining how its recom- 
mendations will be implemented. Where committee recommendations are not 
accepted or the institution is not able to implement them, the reasons for those 
decisions should also be discussed. 

Follow-up 

Since most improvements to graduate programs can be made only over a period of 
time, it is essential to establish a procedure for follow-up to ensure that the 
memorandum of understanding has been implemented. Approximately one year 
after the memorandum is signed, or at some other agn^d-upon date, the parties 
responsible for each portion of the implementation should be asked to report on what 
has been accomplished. If the task is not completed, the relevant faculty and 
administrators should agree on any additional action to be taken and on a timetable 
for its completion. This process should be repeated until the implementation is 
complete, with the final evaluation coming at the next scheduled program review. 

Depending on the structure of the process and the content of the recommendations, 
several possible models may be appropriate for follow-up: the use of an external 
committee or single review er; one or more meetings of the department chair and 
relevant deans; review by a standing university committee or panel; review by the 
graduate council; and others. Whatever model is chosen, it must be one that will 
ensure that action will be taken — action which has the support of the administration 
and the confidence of the faculty. 
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Program review is n crucial part of institutional planning and development. It 
documents the achievements of faculty and students in an academic program, and 
indicates areas where even greater effort is justified. 

This document has attempted to present the critical components of a graduate 
program review. Among the more important principles stressed here are: that the 
review may be an internal and/or external process, that it involve evaluations of the 
quality of the program and the adequacy of resources, that it is forward-looking and 
objective, and that it result in action. 

The role of the graduate dean in program review — whether the reviews arc 
graduate-only or comprehensive — is clear and central. If the reviews are of graduate 
programs exclusively, then the dean should participate in meeting with the review 
committee and/or external consultants and receiving and implementing the final 
report. If the review covers both undergraduate and graduate programs, the 
graduate dean should be involved at the campus-wide level, foi nutating the final set 
of recommendations on the basis of the review process. 

The academic review of graduate programs is a time-consuming, costly, and 
exhausting process. It is also enlightening, stimulating, and gratifying. When done 
well, it provides an impetus for change and strengthens the quality of graduate 
programs. Most important of all it reaffirms for the academic community and the 
public alike the inherent values 01 higher education; intellectual honesty, collegiality, 
and excellence in the pursuit of knowledge. 



ERIC 



3D 




ppendix 



Resources for Conducting Program Reviews 

Graduate Program Self-Assessment Service: Sponsored by CGS and the GRE Board, 
this service provides questionnaires to institutions to review masters or doctoral 
programs. Separate questionnaires are available for current students, current faculty, 
and graduates of the program. ETS will provide questionnaires, compile the results, 
and conduct a sophisticated analysis for the requesting institution, based on a 
predetermined fee. Or, an institution may purchase the questionnaires directly and 
conduct its own analysis. All of the questions are multiple choice; however, there is 
an opportunity for the institution to add questions of its own. For more information, 
please contact the: 

Graduate Program Self-Assessment Service 
Educational Testing Service 
Box 2878 

Princeton, NJ 08541 
(609) 734-5966 

CGS Consultation Service; The Council of Graduate Schools, through its Consulta- 
tion Service, will arrange for one of more consultants experienced in the review of 
graduate programs to provide expertise on the criteria for and process of program 
review. These individuals are usually graduate deans or other senior administrators 
who are asked to answer specific questions or to develop an entire review process. 
The requesting institution pays a modest honorarium plus expenses. For more 
information, please refer to the brochure on the CGS Consultation Service or contact 



Council of Graduate Schools 
One Dupont Circle, N.W., Suite 430 
Washington, D.C 20036 
(202) 223-3791 

The Higher Education Director}/ is published every year and contains, in addition 
to institutional listings, a list of many of the professional associations in the various 
disciplines in the United States. Administrators looking for the names of possible 
external reviewers for a graduate program might want to contact the televant 



the: 



ERIC 



31 




professional association(s) for suggestions. Individual departments, of course, can 
provide information on their professional associations as well. The Director]! is 
available from: 

Higher Education Publications, Inc. 
2946 Sleepy Hollow Road 
Falls Church, VA 22044 
(703) 532-2300 
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